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About Success Boston 


Success Boston is Boston’s citywide college completion initiative. Together, the Boston Foundation, the Boston 
Public Schools (BPS), the City of Boston, the Boston Private Industry Council (PIC), 37 area institutions of 
higher education, led by UMass Boston and Bunker Hill Community College, and local nonprofit partners are 
working to double the college completion rate for students from the BPS. Success Boston was launched in 2008 in 
response to a longitudinal study by Northeastern University and the PIC, which showed that only 35% of those 
BPS graduates who had enrolled in college ever completed a postsecondary certificate, Associate’s or Bachelor’s 
degree within seven years of graduation from high school. Success Boston’s theory of change is that cross-sector 
partnerships, guided by data and mutual accountability, will significantly change the post-secondary trajectory 
for BPS graduates. Its strategic framework focuses on helping Boston’s high school students “Get Ready, Get In 
and Get Through” college, then “Get Connected” to jobs. A core intervention of Success Boston’s is a transition 
coaching model; other key activities within this framework include improving academic preparation and offering 
as-needed supports through higher education institutions until students successfully attain a degree prepared to 
enter the workforce. In 2014, the Boston Foundation received a grant from the Corporation for National and 
Community Service to expand this effort. This $6 million Social Innovation Fund award gave the Foundation the 
resources necessary to expand Success Boston’s transition coaching model from serving 300 to 1,000 students from 


each of the Boston Public Schools classes of 2015, 2016, 2017 and 2018. 


About Abt Associates 


Founded in Cambridge, Massachusetts in 1965, Abt provides applied research and consulting services to 
government agencies, philanthropic, nonprofit, and commercial organizations around the world. Abt’s mission is 
to improve the quality of life and economic well-being of people worldwide. It applies its exceptional subject matter 
expertise, outstanding technical capabilities in applied research, and strategic planning to help local, national and 


international clients make better decisions and deliver better services. 
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This report is based upon work supported by the Social Innovation Fund (SIF) which unites public and private 
resources to evaluate and grow innovative community-based solutions with evidence of results. The Social 
Innovation Fund was a program of the Corporation for National and Community Service that received funding 
from 2010 to 2016. Using public and private resources to find and grow community-based nonprofits with 
evidence of results, SIF intermediaries received funding to award subgrants that focus on overcoming challenges in 


economic opportunity, healthy futures, and youth development. 
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Endnotes 


Introduction 


Access to middle-class jobs increasingly requires a college degree or credential. College graduates earn 
more, are less likely to suffer job losses in a recession, and are projected to have superior long-term labor 
market prospects.’ In Boston, the college graduation rate for the city’s public high school graduates increased 
from 39 percent in 2000? to 52 percent in 2011.° Nevertheless, challenges remain. The graduation rate is 

likely still not sufficient to meet the predicted demand for a college-educated workforce. Furthermore, 
students from low-income households, first-generation students, and racial/ethnic minorities are typically 
underrepresented in postsecondary education.* 


The Success Boston initiative was launched in 2008 specifically to improve Boston public high 

school graduates’ access to and success in college, particularly for members of groups traditionally 
underrepresented in college degree attainment. Higher college graduation rates, in turn, are expected to 
increase students’ access to employment in such local industries as technology, financial services, higher 
education, and health care that routinely require advanced training. 


The Success Boston initiative unites the Boston Foundation, City of Boston, Boston Public Schools (BPS), 
University of Massachusetts Boston, Bunker Hill Community College, other regional colleges and 
universities, and local nonprofit organizations in a concerted effort to improve college completion rates 
for Boston public school graduates. The partners focus on boosting college persistence and ultimately 
improving college completion rates for these graduates through a purposeful combination of program, 
policy, and practice-based activities. The Boston 
Foundation is the convening “backbone” 

Who provided Success organization of the Success Boston initiative; it 
Boston Coaching services? funds and supports the nonprofit organizations that 


Local nonprofit organizations provided provide one of the core programs within the overall 


one-on-one coaching to students for their first 
two years in college (during the time of this 
study), including: 


initiative: one-on-one transition coaching, or Success 
Boston Coaching (SBC). 


SBC is designed to support recent high school 
Boston Private Industry Council graduates entering college who are most likely to 
Bottomilane leave before completion—that is, first-generation 
college and low-income students of color. Through 
College Bound Dorchester ‘ 
this program, students can access one-on-one 

Freedom House coaching, starting as early as the last year of high 
Hyde Square Task Force school and continuing for the first two years after 


Gaia high school graduation. 


The Steppingstone Foundation Since the Success Boston initiative’s start in 2008, 


West End House 


SBC has been offered to 10 successive cohorts of 


Boston public high school graduates, starting with 
Additionally, uAspire provided financial the class of 2009. In the fall of 2015, and funded 
aid assistance and advising to students and 


in part through a Social Innovation Fund (SIF) 
training for Success Boston coaches. 


grant from the federal Corporation for National 


and Community Service, the Boston Coaching for 
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Completion scale-up broadened the reach of the SBC program from several hundred to 1,000 Boston young 
adults per cohort. In the first two years after the scale-up, eight nonprofit organizations provided coaching 
to 2,053 students attending more than 50 colleges, with 53 percent of these students attending Bunker Hill 
Community College and University of Massachusetts Boston alone. These nonprofit organizations, partner 
colleges, and the Success Boston network collaborate to provide transition coaching to support students on 
the path to college graduation (see Exhibit 1). 


EXHIBIT 1 
Success Boston Coaching model 
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SBC coaches support students throughout the academic year and summer, offering proactive and 
on-demand guidance to help prepare students to navigate, increasingly independently, through to college 
graduation.° They provide support on life skills, study skills, help-seeking strategies, and academic 
skills; they help students develop meaningful relationships, clarify goals, access networks, understand 
college culture, and make college life feasible; and they provide job and career mentoring. Partner colleges 
communicate with coaches and help coordinate coaching services on their campuses. The SBC network, 
which the Boston Foundation oversees, facilitates communication across organizations; it also provides 
coaches access to specialized training about financial aid from uAspire, a national nonprofit organization, 
as well as access to training on other topics. 


Earlier studies have demonstrated that coaching increases student persistence and academic achievement 
in college.® There is also descriptive evidence that for some students, coaching may lead to higher college 
completion rates.’ This SBC study aims to build on prior work by examining both the implementation and 
impact of SBC after the scale-up. The sections that follow provide an overview of the study design, findings, 
and discussion points. 
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Study Design 


Research Questions. This brief highlights findings from the Success Boston The Story of Scaling Up: Impact 
of Success Boston's Coaching to Completion report, which answers two key research questions:® 


1. What is the effect of SBC on student success in college? 


2. How, if at all, do the impacts of SBC vary by student characteristics and features of the coaching? 


Sample. The study includes students who: 
m Enrolled in college in the fall after high school graduation (either fall 2015 or fall 2016); 


@ Enrolled in a college in which at least one SBC student and at least one potential comparison student 
were enrolled in that given year; and 


= Had no missing information on key baseline characteristics (free and reduced-price lunch status and 
high school GPA) used in the matching process. 


The study sample includes up to 1,480 SBC students and 8,978 non-SBC students; the specific number of 
students varies by outcome due to data availability. 


Analysis. Using a quasi-experimental design, this study compares outcomes for the students who 

§4q4 P & y P 
participated in SBC (“treatment” group) to outcomes of a group of similar students (“comparison” group) 
who did not participate. 


Because the study uses a rigorous quasi-experimental design, it provides causal evidence that observed 
differences in outcomes between coached and similar non-coached students are due to participation in SBC. 


The study also considers whether any impacts vary as a function of particular student characteristics 
or features of coaching. This brief highlights the results by one key feature of coaching, the frequency of 
interactions—the remaining analyses can be found in the full report. 


Data Sources. The study team collected data from BPS, the Massachusetts Department of Elementary 
and Secondary Education, and 11 partner colleges, and the SBC program database. 


Outcomes. The outcomes for this study fall into three domains:? 

1. Persistence (as measured by persistence into the second year and persistence into the third year); 
2. Academic achievement (as measured by cumulative GPA); and 

3. Financial aid (as measured by Free Application for Federal Student Aid (FAFSA) renewal). 


The outcomes above are the focus of this brief. However, the full report also looks at three additional 
outcomes: full-time status, semesters enrolled in non-credit bearing courses, and credit accumulation. 
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Findings 


Summar 
y «= PERSISTENCE SBC coached 


students are 5% more likely than are 
non-coached peers to persist into the second 
year of college. 


The analyses estimate that SBC students have 
better early college outcomes than do their 
carefully matched peers not participating in 
SBC. Specifically, SBC students are more likely 
to persist into their second year of college (82.6 


ACHIEVEMENT After two years of 


percent for SBC students in the treatment group 
vs. 78.3 percent for the non-coached students college, coached students had 
in the comparison group). Additionally, SBC comparable cumulative GPAs to non-coached 


students are more likely to complete FAFSA students. 
renewals for their second year of college (83.9 


&=s~, FINANCIAL AID Coached students are 
8% more likely to renew their FAFSA 
than non-coached students. 


percent vs. 77.7 percent). 


Persistence into the Second and Third Years of College 


° A key expectation of the SBC model is that students are more likely to persist in college as 

a coaches help them navigate and manage the academic, financial, and social-emotional challenges 

= typically faced by beginning college students. To test this hypothesis, this study uses two primary 
outcome measures to assess persistence: persistence into the second year of college and persistence into the 
third year of college.’” These persistence outcomes measure whether students who enrolled in college after 
their high school graduation returned to college in the fall of succeeding academic years. Persistence into 
the second year of college indicates whether students in the 2015 and 2016 cohorts enrolled in the fall semester 
of 2016 and 2017, respectively. Persistence into the third year of college reflects whether students in the 2015 
cohort also enrolled in the fall semester of 2017. 


Key to Impact Exhibits 


* |mpact is significant at the 5 percent level Persistence outcomes come from NSC data provided 
** Impact is significant at the 1 percent level by Boston Public Schools and Massachusetts 
Department of Elementary and Secondary Education and 


+ Significance determined with the Benjamini- college administrative data 


Hochberg correction. 
Academic and FAFSA outcomes come from college 
administrative data 
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EXHIBIT 2 
Impact of SBC on persistence into second and third years of college 


78.3% 
Persistence 
into 2nd Year+ 
82.6% 


: 71.9% 
Persistence 
into 3rd Year+ 
74.7% 


0 20% 40% 60% 80% 100% 


Percentage of Students 


BM Adjusted Comparison Group Mean Hi Treatment Group Mean 


Note: N=5,863 for overall sample (N=1,234 for treatment and N=4,629 for comparison) for persistence into the second year. 
N=2,719 for overall sample (N=561 for treatment and N=2,158 for comparison) for persistence into the third year. 


The top panel of Exhibit 2 above shows that 82.6 percent of SBC students in the treatment group and 78.3 
percent of non-coached students (comparison group) persisted into the second year of college. The 4.3 
percentage point impact of SBC on persistence is positive and statistically significant. This percentage point 
change is the equivalent of a 5 percent increase in persistence rates. Both groups persisted into the second 
year at higher rates than the 73.9 percent national average for students starting college in the fall of 2016 
(NSC 2018). 


The bottom panel in Exhibit 2 shows that the impact of SBC on persistence into the third year is not 
significant. (This third-year outcome is measured only for the 2015 cohort, so it should be interpreted with 


caution due to the smaller sample size.)" 


PERSISTENCE IS AN IMPORTANT The positive findings across the persistence outcomes are 
SUCCESS INDICATOR: STUDENTS noteworthy for several reasons. First, these corroborate prior 


MUST PERSIST IN ORDER TO findings in the Success Boston interim impact report (Linkow et 
COMPLETE. al. 2017b) as well as research conducted by Sum, Khatiwada, and 
Palma (2014), which found evidence of a positive and statistically 
significant effect of Success Boston Coaching on college persistence. Second, the findings are consistent with 
studies of other coaching interventions shown to improve students’ likelihood of persisting in college (e.g., 
Bettinger and Baker 2014). And third, persistence is an important success indicator; students must persist in 


order to complete. 
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Achievement 


Wa Another key hypothesis of the SBC study is that coaching supports can help students manage 
various academic challenges that they may face in college, ranging from difficult coursework to 
course selection and time management. By helping students access available campus supports 
and cope with stressors related to the academic demands of college, coaches can potentially help students 


improve their academic achievement. 


The analyses described below examine the impact of SBC on one primary outcome measure of academic 
achievement: cumulative GPA.'? Cumulative GPA is drawn from students’ most recent semester enrolled 
and corresponds with the following values: A=4.0, B=3.0, C=2.0, D=1.0. 


Exhibit 3 illustrates that SBC students in the treatment group have comparable cumulative GPAs, on 
average, relative to non-coached students in the comparison group. Students in both the treatment and 
comparison groups had GPAs that fell in the C/C+ range (approximately 2.4 points on a four-point scale). 


EXHIBIT 3 
Impact of SBC on cumulative GPA 


Cumulative GPA 


0 1 2 3 
Grade Point Average (GPA) 


BB Adjusted Comparison Group Mean Hl Treatment Group Mean 


Note: N=3,553 for overall sample (N=814 for treatment and N=2,739 for comparison). 
Taken as cumulative GPA from the most recent GPA data available. 
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Financial Aid 


One of the goals of SBC is to help students navigate the financial aid process and presumably help 
reduce the gap between the cost of college and what students can actually afford. Helping students 

to complete the FAFSA is a key part of this process. In fact, uAspire offers training for SBC coaches 
ona range of financial aid and planning topics and hosts FAFSA completion events for SBC students. In 
2016-2017, some 67 percent of SBC students also received text message nudges from uAspire advisors. To this 
end, the SBC study examines FAFSA renewal, measuring whether students complete and submit a FAFSA 
renewal application for their second year of college, unlocking access to federal student aid. SBC serves 
largely low-income, Pell Grant—eligible students; completing the FAFSA is crucial for these students to access 


the financial aid they need to succeed in college. 


A large majority of both SBC and non-coached students renewed their FAFSA for their second year of 
college, reflecting the high proportion of students who are eligible for federal financial aid in the study 
sample (74 percent of SBC students and 75 percent of those not coached were eligible for free/reduced-price 


lunch in high school). 


SBC students in the treatment group renewed their FAFSA at a higher rate than the students in the 
comparison group, and the difference in renewal rates (6.2 percentage points) is positive and statistically 
significant (Exhibit 4). Among the treatment group students, 83.9 percent renewed their FAFSA for their 
second year of college versus 77.7 percent of comparison students. This percentage point change is the 
equivalent of an 8 percent increase in FAFSA renewal rates. 


EXHIBIT 4 
Impact of SBC on FAFSA renewal 


FAFSA Renewal 
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Percentage of Students 
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Note: N=3,597 for overall sample (N=821 for treatment and N=2,776 for comparison). 
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Differences in Impacts by Frequency of Coaching 


In addition to examining program impacts, the study also considers whether any impacts vary 
as a function of particular student characteristics or features of coaching. In the full report, 


differences in coaching impacts were examined by four types of student characteristics: gender, 
underrepresented minority status, high school GPA, and two- vs. four-year college enrollment status. It 
can be useful to examine differences by student characteristics when making decisions about the coaching 
program. For example, if a particular group of students may be most in need of coaching, or if coaching 

is shown to be particularly helpful for certain groups of students, program administrators may consider 
targeting the coaching at those groups. However, because the study did not find any consistent patterns 

of differences by student characteristics, this section focuses on the question of whether impacts vary by 


coaching feature. 


This brief highlights the results by one key feature of coaching: the frequency of interactions. The remaining 
analyses can be found in the full report. Frequency of interactions measures the number of interactions 
across all modes (in-person, text, email, phone, and social media) between students and their coach across 
the same two-year time period as described above. To simplify analyses, two groups of students were 
created based on the median value: Those who fell into the “low” group had 14 interactions or fewer and 


those who fell into the “high” group had more than 14 interactions per academic year. 


Findings show that SBC students have more positive college outcomes when they experience more 
interactions (Exhibit 5). Further, the positive impacts of coaching are concentrated among students 


EXHIBIT 5 
Impact of SBC by frequency of coach-student interactions 
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Persistence into Persistence into Cumulative GPA FAFSA Renewal 
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Note: For each outcome, there is a statistically significant difference between the impact of coaching for students experiencing 
a high number of interactions and the impact of coaching for students experiencing a low number of interactions. 
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who experience more frequent coach interactions. 


ON THE WHOLE, THESE RESULTS These results suggest that coaching is more 
SUGGEST THAT COACHES FREQUENTLY effective for students who experience more than 
CHECKING IN WITH STUDENTS MAY 14 interactions throughout the academic year. 

BE MORE IMPORTANT FOR STUDENTS’ 

POSTSECONDARY SUCCESS THAN By contrast, when we looked at how impacts vary 


MEETING FOR LONGER PERIODS OF TIME. by duration of coaching interaction, we found that 


longer coaching interactions were no more beneficial 

than shorter coaching interactions, except for in the 
case of renewing the FAFSA. Longer interactions may be necessary for completion of the FAFSA to allow 
time for coaches to work through the entire application with students. On the whole, these results suggest 
that coaches frequently checking in with students may be more important for students’ postsecondary 
success than meeting for longer periods of time. 
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Discussion 


Success Boston coaches support students to successfully navigate college, through both referrals to campus 
resources and direct support, across areas that research suggests can be helpful in improving student 
outcomes. Coaches connect students to resources such as tutoring and financial aid services, help them 
plan their coursework and identify a major, work with them to understand financial aid processes and 
options, provide them with time management and study strategies, and develop positive relationships 


with students.” 


CORE TO THE SBC MODEL ARE 
Perhaps uniquely so, the SBC program integrates these different RESPONSIVENESS TO STUDENT 


features into a single intervention that is tailored to individual NEEDS AND CUSTOMIZATION 
student needs. Core to the SBC model are responsiveness to OF SUPPORT. 

student needs and customization of support to address each 

student’s individual requirements for support and contact. Coaches vary the frequency, modes of 
communication, and focus of supports to “meet students where they’re at.”"* Though one coach may 
provide support across a variety of topics, any individual student may not receive support on all topics, 


depending on that student’s needs and interests. 


The expansion of SBC did not come at the expense of how often coaches interact with students. Students in 


the scaled-up program typically interacted with their coaches between one and two times a month during 


EXHIBIT 6 
Trends in student persistence by cohort and coached status 
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the academic year and each in-person meeting was about 30 minutes, both of which were similar to how 
often and how long coaches interacted with students in the earlier cohorts.® 


The effects of the scaled-up SBC program can be characterized as positive and moderate in magnitude. 
Yet the impacts found for the program at scale are smaller than those seen for earlier cohorts.'° Exhibit 6 
compares the persistence rates of coached students (purple circles) and comparison students (red squares) 
for each set of cohorts (2013 and 2014 cohorts are the dashed lines and 2015 and 2016 cohorts are the solid 
lines). It may be that the shrinkage in the impact of SBC is caused by improvements in the outcomes of 
comparison group students (see the lines with red squares in Exhibit 6). There is some evidence that the 
colleges participating in SBC, in particular those that enroll large proportions of SBC students, have made 
changes in recent years to improve their student support services. For example, some of the colleges have 
established their own cadre of “coaches” to support students coming from outside of Boston; other colleges 
have centralized their non-academic support services to improve accessibility to students. Overall, coached 
and non-coached students from the 2015 and 2016 cohorts (solid lines) are persisting at slightly higher rates 
than students from the 2013 and 2014 cohorts (dashed lines). College 


changes could be improving outcomes for all students. 


THE IMPACT RESULTS 


SUGGEST THAT THE 

SBC MODEL HAS Even with overall increases in student outcomes, SBC continues to 
MEANINGFUL AND add value to participating students. The impact results suggest that 
SIGNIFICANT PROMISE. the SBC model has meaningful and significant promise. A model in 


which professional coaches deliver sustained, proactive, and responsive 
support within a larger education network may not necessarily need to be “high touch” to accomplish the 
overarching program goal: to move more students more quickly and effectively along the pathway to college 
completion. The proactive, sustained, and adaptive advising that is Success Boston Coaching can improve 
the college outcomes of students who are typically underserved and underrepresented in postsecondary 


education. 
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Two domains had multiple outcomes: persistence and achievement. For these domains we distinguish 
between primary and exploratory outcomes. Primary outcomes are those most closely related to the theory 
of change—and are the focus of this brief. See the full report for a comprehensive discussion of exploratory 
outcomes: tbf.org/scalingup-report. Exploratory outcomes include: full-time status, semesters enrolled in 
non-credit-bearing courses, and credit accumulation. 


As described in the full report, SBC students were also slightly more likely to be enrolled full-time relative 
to non-coached students in the comparison group. 


Because two primary outcomes within the single domain of persistence are tested, statistical significance 
tests use the Benjamini-Hochberg correction. 


The full report also examines impacts on credit accumulation, showing that SBC students also earned 
slightly more credits toward their degrees than the comparison group students. 
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Linkow et al. 2017a. 
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